SignGD with error feedback meets lazily aggregated technique: Communication-efficient algorithms for distributed learning

The proliferation of massive datasets has led to significant interests in distributed algorithms for solving large-scale machine learning problems. However, the communication overhead is a major bottleneck that hampers scalability systems. In this paper, we design two communication-efficient tasks. first one named EF-SIGNGD, which use 1-bit (sign-based) gradient quantization method save bits. Moreover, error feedback technique, i.e., incorporating made by compression operator into next step, employed convergence guarantee. second algorithm called LE-SIGNGD, introduce well-designed lazy aggregation rule EF-SIGNGD can detect gradients with small changes and reuse outdated information. LE-SIGNGD saves costs both transmitted bits rounds. Furthermore, show convergent under some mild assumptions. effectiveness proposed demonstrated through experiments on real synthetic data.